Designing a New Bloom Filter-based Index for Distributed Data Management ⋆

Distributed architectures are widely used in Internet applications nowadays. In such systems, one of the key techniques is how to maintain an indexing data structure which records elements of each single node in the system. Bloom filter is one of the popular solutions. The beautiful mathematical format offers a fast and space-efficient solution for probabilistic membership presentation. In many Internet applications, user access for items follows Zipf’s law where a small number of items attract many visits. According to that phenomenon, we propose a selective insertion method of bloom filter to reduce the workload of BFs by finding an optimal load ratio. The experiments show that our new approach can reduce the false lookup time by 36% compared with the pure bloom filter approach.